Bellman Equation Research Articles

An automated multi-layer perceptron discriminative neural network based on Bayesian optimization achieves high-precision one-source single-snapshot direction-of-arrival estimation

This paper proposes an innovative global solution which is a pioneering work applying automated machine learning algorithms to remarkable precision sparse underwater direction-of-arrival (DOA) estimation that views the subaquatic sparse-sampling DOA estimation problem as a classification prediction task. The proposed solution, termed automated multi-layer perceptron discriminative neural network (AutoMPDNN), is built upon a Bayesian optimization framework. AutoMPDNN transforms sparsely sampled time-domain signals into the complex domain, preserving essential components in a one-source single-snapshot scenario. Leveraging Bayesian optimization principles, the algorithm embeds necessary hyperparameters into the loss function, effectively defining it as a maximum likelihood problem using the upper confidence bound function and incorporating sparse signal features. We also explore the model space architecture and introduce variants of AutoMPDNN, denoted as AutoMPDNNs_ln (n = 2,3,4). Through a series of plane wave simulation experiments, it is demonstrated that AutoMPDNN achieves the highest prediction performance for one-source single-snapshot scenarios compared to classical DOA estimation algorithms that incorporate sparse representation approaches, as well as contemporary deep learning DOA methods under varying conditions.

Optimal control of stochastic differential equations with random impulses and the Hamilton–Jacobi–Bellman equation

AbstractIn this article, we study the optimal control of stochastic differential equations with random impulses. We optimize the performance index and add the influence of random impulses to the performance index with a random compensation function. Using the idea of stochastic analysis and dynamic programming principle, a new Hamilton–Jacobi–Bellman (HJB) equation is obtained, and the existence and uniqueness of its viscosity solution are proved.

Synthesis of the european alara network 20th workshop "alara for interventional radiology and nuclear medicine".

The European ALARA Network regularly organises workshops on topical issues in radiation protection. The topic of the 20th workshop was: "ALARA for interventional radiology (IR) and nuclear medicine (NM)". The main objective was to examine the challenges faced when applying the optimisation principle (ALARA) in IR and NM and to consider how ALARA could be better implemented for patient and staff exposures. This memorandum provides a synthesis of the workshop sessions and recommendations arising from the working groups discussion. Parallels are drawn with the recommendations coming from the 13th EAN workshop on "ALARA and the medical sector" (2011) to consider how the optimisation challenges in IR and NM have evolved over the past decade.
Current levels of exposure are presented along with operational practice and the challenges and opportunities for improvement, both in monitoring and practice. Whilst RP challenges remain, the application of ALARA appears more established in IR compared with experiences reported in 2011. The application of ALARA to emerging technologies in the NM setting is in need of further development to ensure that radiation protection is considered at all stages in the development process of new radiopharmaceuticals. Besides technical and operational aspects, the importance of education and training, human factors and , broadly, the radiation protection 'culture' were deemed fundamental to the success of the application of ALARA and where further emphasis is needed. All concerned parties, MPEs, RPEs, medical staff, manufacturers and regulators have a role to play in the application of ALARA and this is discussed in the memorandum. 
Many of the recommendations from the 13th EAN workshop remain applicable today and overlap with the recommendations arising from the 20th workshop. This should prompt attention given that the use of IR and the development of novel radiopharmaceuticals for NM is only anticipated to increase with time.

Scalable computation of energy functions for nonlinear balanced truncation

Nonlinear balanced truncation is a model order reduction technique that reduces the dimension of nonlinear systems in a manner that accounts for either open- or closed-loop observability and controllability aspects of the system. A computational challenges that has so far prevented its deployment on large-scale systems is that the energy functions required for characterization of controllability and observability are solutions of various high-dimensional Hamilton–Jacobi–(Bellman) equations, which are computationally intractable in high dimensions. This work proposes a unifying and scalable approach to this challenge by considering a Taylor-series-based approximation to solve a class of parametrized Hamilton–Jacobi–Bellman equations that are at the core of nonlinear balancing. The value of a formulation parameter provides either open-loop balancing or a variety of closed-loop balancing options. To solve for the coefficients of Taylor-series approximations to the energy functions, the presented method derives a linear tensor system and heavily utilizes it to numerically solve structured linear systems with billions of unknowns. The strength and scalability of the algorithm is demonstrated on two semi-discretized partial differential equations, namely the Burgers and the Kuramoto–Sivashinsky equations.

The Effect of Utilizing Gamification in Enhancing English Language Skills in University Settings

This article aims to explore the effect of applying gamification in English language skills as a pedagogical approach to enhance students’ academic performance in university settings. The researchers employed a mixed-method approach to examine the impact of gamification on English language skills. Gamification involves integrating game elements and mechanics into non-game contexts, such as educational activities. The article examined the effect of gamification and its implementation in teaching English language skills in university settings. The study finds out that gamification offers a promising avenue for promoting engagement, motivation, and proficiency in teaching English language skills. However, further research is needed to investigate gamified learning environments' long-term effects and optimal design principles.

Finite Element Analysis and Design of a Flexible Thermoelectric Generator with a Rhombus Gap Structure.

Flexible thermoelectric generators (f-TEGs) offer an opportunity to realize wearable, self-powered electronic devices. A typical f-TEG consists of flexible electrodes and rigid thermoelectric (TE) legs in a flexible package. In the realm of f-TEGs utilizing flexible electrodes and TE cuboids, our unwavering objective lies in the attainment of enhanced flexibility and elevated energy conversion efficiency. In this paper, we employ a quasi-three-dimensional thermal model to design an f-TEG with a rhombus gap structure (E/A-RhTEG) with its optimized performance validated by simulation and experiment. Additionally, the lateral and vertical thermal resistances are introduced to further explain the optimizing principle in the f-TEG's output performance. Compared with the conventional TEG with a rectangular air gap structure (E/A-ReTEG), E/A-RhTEG demonstrates improved energy conversion efficiency to some extent. Simulation results indicate that the output power and energy conversion efficiency of a 25-np-pair E/A-RhTEG at a 30 K temperature gradient reach 8.45 mW and 2.55%, which represent a performance improvement of 3.09 and 6.28%, respectively, compared to E/A-ReTEG. To further elucidate the optimization principle in the performance of f-TEGs, additional considerations are given to the lateral and vertical resistances. In this study, E/A-RhTEG comprising 25 np pairs is fabricated utilizing TE cuboids. Experimental findings indicate that E/A-RhTEG exhibits a voltage output of 127.07 mV when subjected to a temperature difference of 30 K, which demonstrates a performance enhancement of 4.06% compared to E/A-ReTEG. Furthermore, this study also demonstrates its implementation when wrapped around a curved surface and successfully achieves a self-powered device system after device performance optimization.

Adaptive neural event‐triggered near‐optimal control for affined uncertain nonlinear discrete‐time system

AbstractA novel event‐triggered heuristic dynamic programming (HDP) algorithm is proposed for the near‐optimal control of uncertain discrete‐time nonlinear input‐affine systems. Based on input‐to‐state stability (ISS) analysis, a new event‐triggered mechanism (ETM) is designed. Under constant coefficients, a Lipschitz‐like assumption that forms the basis of the event‐triggering condition is considered to be conservative. To further reduce the conservativeness of the triggering condition and enlarge the average interevent time, an adaptive threshold parameter is utilized in the proposed ETM. In the HDP algorithm framework, model, critic, and action network are adopted to achieve state estimation, approximation to the optimal cost function, and solution to Hamilton–Jacobian–Bellman (HJB) equation. Under the proposed event‐triggered HDP algorithm, the closed system is proved to possess semiglobal uniform ultimate boundedness (SGUUB). Finally, by conducting simulation, it shows that on the premise of satisfying control performance, the event‐triggered strategy can realize reduction on the updating frequency of the controller.

Mechanism design for public projects via three machine learning based approaches

We study mechanism design for nonexcludable and excludable binary public project problems. Our aim is to maximize the expected number of consumers and the expected agents’ welfare. We first show that for the nonexcludable public project model, there is no need for machine learning based mechanism design. We identify a sufficient condition on the prior distribution for the existing conservative equal costs mechanism to be the optimal strategy-proof and individually rational mechanism. For general distributions, we propose a dynamic program that solves for the optimal mechanism. For the excludable public project model, we identify a similar sufficient condition for the existing serial cost sharing mechanism to be optimal for 2 and 3 agents. We derive a numerical upper bound and use it to show that for several common distributions, the serial cost sharing mechanism is close to optimality. The serial cost sharing mechanism is not optimal in general. We propose three machine learning based approaches for designing better performing mechanisms. We focus on the family of largest unanimous mechanisms, which characterizes all strategy-proof and individually rational mechanisms for the excludable public project model. A largest unanimous mechanism describes an iterative mechanism, which is defined by an exponential number of mechanism parameters. Our first approach describes the largest unanimous mechanism family using a neural network and training is carried out by minimizing a cost function that combines the mechanism design objective and the constraint violation penalty. We interpret the largest unanimous mechanisms as price-oriented rationing-free (PORF) mechanisms, which enables us to move the mechanisms’ iterative decision making off the neural network, to a separate simulation process, therefore avoiding the vanishing gradient problem. We also feed the prior distribution’s analytical form into the cost function to achieve high-quality gradients for efficient training. Our second approach treats the mechanism design task as a Markov Decision Process with an exponential number of states. During the Markov decision process, the non-consumers are gradually removed from the system. We train multiple neural networks, each for a different number of remaining agents, to learn the optimal value function on the states. Training is carried out by supervised learning toward a set of manually prepared base cases and the Bellman equation. Our third approach is based on reinforcement learning for a Partially Observable Markov Decision Process. Each RL episode randomly draws a type profile, which is hidden from the RL agent (mechanism designer). The RL agent only observes which cost share offers have been accepted under the largest unanimous mechanism under discussion. We use a continuous action space reinforcement learning approach to adjust the offer policy (i.e., adjust mechanism parameters). Lastly, our first two approaches use “supervision to manual mechanisms” as a systematic way for network initialization, which is potentially valuable for machine learning based mechanism design in general.

Optimized Backstepping Cooperative Control for Output-Constrained Stochastic Nonlinear Network Systems via a Multibridge-Hole Function.

In this article, a new leader-following tracking control approach is investigated for stochastic multiagent systems with multibridge-hole output constraints. The multibridge-hole output constraints mean that the output of the system is constrained in some intervals and unconstrained in other intervals. The constrained and unconstrained intervals can be set arbitrarily. By designing a new shift function to construct the barrier Lyapunov function, the optimal controller is constructed by combining the backstepping technique with the adaptive dynamic programming technique. The model network is used to estimate the unknown disturbances and uncertainty terms in the system. The critic network and the actor network are constructed such that the designed controller adheres to the Bellman optimality principle and gives the optimal solution of the system. The proposed control method is versatile and compatible with various types of output constrained control problems, such as unconstrained control problems, constrained control problems, and delay constrained problems without changing the structure of the controller. Finally, some simulation results are given to verify the effectiveness of the method.

MÜASİR TƏHSİL PARADİQMASI KONTEKSTİNDƏ TƏLİM PROSESİNİN SƏMƏRƏLİLİYİNİN OPTİMALLAŞDIRILMASI

Optimizing the Efficiency of the Training Process in the Context of the Modern Educational Paradigm Summary The article examines the problem of increasing the efficiency of the modern educational paradigm through the optimization of the educational process as the main tool for the formation of personality according to the abilities and needs of students. The principles of optimizing education are described, the principle of optimality is mentioned, and pedagogical conditions that optimize the teaching process are discussed. The article also defines the optimality criteria for determining the efficiency level of the educational process, the principles of optimizing the educational and educational process, which are necessary to create favorable conditions in the optimization process. Key words: optimization, educational paradigm, efficiency, principles, pedagogical values

A comparison principle for semilinear Hamilton–Jacobi–Bellman equations in the Wasserstein space

A comparison principle for semilinear Hamilton–Jacobi–Bellman equations in the Wasserstein space

Differential graphical game‐based multi‐agent tracking control using integral reinforcement learning

AbstractThis paper studies the cooperative tracking control problem of interacted multi‐agent systems (MASs) under undirected communication. Based on differential graphical game theory, the MAS tracking control problem is formulated as an infinite horizon cooperative differential graphical game‐theoretic tracking control framework, where a multi‐objective optimization problem is designed and then cast into a Pareto‐equivalent single‐objective optimization problem using a scalarization method. Necessary and sufficient conditions for the existence of the Pareto‐optimal strategy to the game theoretic tracking control are established, where it has been proven that the solution to the integral Bellman optimality equation leads to Pareto‐optimal strategy. Then, an off‐policy integral reinforcement learning scheme to find optimal control strategy using a pure data‐driven manner is developed, which consumes less computation efforts than the traditional learning scheme. Simulated results are conducted to validate the effectiveness of the proposed game and IRL‐based tracking control method.

Optimization for stochastic model arisen from investment problem associated with default risk

In this paper, we study the optimal strategy arisen from the excess-of-loss reinsurance and asset allocation in defaultable markets under a general stochastic model. By developing the corresponding dynamic programming approach, we establish the optimal investment approach through two sub-problems: a pre-default case and a post-default case, respectively, characterized by the obtained Hamilton–Jacobi–Bellman (HJB) equations. We show the existence of a classical solution to the pre-default case via super-sub solution techniques and give an explicit characterization of the optimal reinsurance and investment policy that maximizes the common used utility associated with the terminal wealth. Verification theorem is further established to show the uniqueness of the corresponding solution of HJB equation that is critical to the desired optimal solution.

Near-Optimal Tracking Control of Partially Unknown Discrete-Time Nonlinear Systems Based on Radial Basis Function Neural Network

This paper proposes an optimal tracking control scheme through adaptive dynamic programming (ADP) for a class of partially unknown discrete-time (DT) nonlinear systems based on a radial basis function neural network (RBF-NN). In order to acquire the unknown system dynamics, we use two RBF-NNs; the first one is used to construct the identifier, and the other one is used to directly approximate the steady-state control input, where a novel adaptive law is proposed to update neural network weights. The optimal feedback control and the cost function are derived via feedforward neural network approximation, and a means of regulating the tracking error is proposed. The critic network and the actor network were trained online to obtain the solution of the associated Hamilton–Jacobi–Bellman (HJB) equation within the ADP framework. Simulations were carried out to verify the effectiveness of the optimal tracking control technique using the neural networks.

Research on an Intelligent Vehicle Trajectory Tracking Method Based on Optimal Control Theory

This study aims to explore an intelligent vehicle trajectory tracking control method based on optimal control theory. Considering the limitations of existing control strategies in dealing with signal delays and communication lags, a control strategy combining an anthropomorphic forward-looking reference path and longitudinal velocity closure is proposed to improve the accuracy and stability of intelligent vehicle trajectory tracking. Firstly, according to the vehicle dynamic error tracking model, a linear quadratic regulator (LQR) transverse controller is designed based on the optimal control principle, and a feedforward control strategy is added to reduce the system steady-state error. Secondly, an anthropomorphic look-ahead prediction model is established to mimic human driving behavior to compensate for the signal lag. The double proportional–integral–derivative (DPID) control algorithm is used to track the longitudinal speed reference value. Finally, a joint simulation is conducted based on MatLab/Simulink2021b and CarSim2019.0 software, and the effectiveness of the control strategy proposed in this paper is verified by constructing a semi-physical experimental platform and carrying out a hardware-in-the-loop test. The simulation and test results show that the control strategy can significantly improve the accuracy and stability of vehicle path tracking, which provides a new idea for future intelligent vehicle control system design.

Differential Games with Additive Payoffs

Differential games with additive payoffs are considered. A new approach to constructing the characteristic function for such games is defined. Properties of optimality principles such as the core and the Shapley value constructed by applying the new characteristic function are studied. The solutions obtained are demonstrated by means of an example.

Minimum loss optimization of flywheel energy storage systems via distributed adaptive dynamic programming

AbstractIn this article, a distributed controller based on adaptive dynamic programming is proposed to solve the minimum loss problem of flywheel energy storage systems (FESS). We first formulate a performance function aiming to reduce total losses of FESS in power distribution applications. Then we use the Hamilton–Jacobi–Bellman (HJB) equation to solve this optimal control problem. The solution of the HJB equation is approximated by neural networks. To achieve distributed control, we estimate the global variables in the HJB equation by using the dynamic average consensus algorithm. A barrier Lyapunov function and a saturation function are introduced to handle the issue of state and input constraints, respectively. Then the stability of the system is proved through the Lyapunov stability analysis. Finally the effectiveness of the proposed strategy is verified by simulations. Simulation results show that FESS can track the power command while minimizing total power losses by interacting with neighbors. The proposed algorithm leads to a loss reduction of compared to the equal power distribution strategy.

Computing perfect stationary equilibria in stochastic games

AbstractThe notion of stationary equilibrium is one of the most crucial solution concepts in stochastic games. However, a stochastic game can have multiple stationary equilibria, some of which may be unstable or counterintuitive. As a refinement of stationary equilibrium, we extend the concept of perfect equilibrium in strategic games to stochastic games and formulate the notion of perfect stationary equilibrium (PeSE). To further promote its applications, we develop a differentiable homotopy method to compute such an equilibrium. We incorporate vanishing logarithmic barrier terms into the payoff functions, thereby constituting a logarithmic-barrier stochastic game. As a result of this barrier game, we attain a continuously differentiable homotopy system. To reduce the number of variables in the homotopy system, we eliminate the Bellman equations through a replacement of variables and derive an equivalent system. We use the equivalent system to establish the existence of a smooth path, which starts from an arbitrary total mixed strategy profile and ends at a PeSE. Extensive numerical experiments, including relevant applications like dynamic oligopoly models and dynamic legislative voting, further affirm the effectiveness and efficiency of the method.

Cobra aerobatic flight for quadrotors using θ-D nonlinear control

The control design for the Cobra maneuver of quadrotors is presented based on quaternion dynamics. Cobra maneuver commands the quadrotor in ± π / 2 (rad) pitch angle such that the conventional hovering assumption for quadrotor flight is not applicable. The finite-time θ-D method is proposed to control the Cobra maneuver. Based on this technique, the intractable Hamilton–Jacobi–Bellman equation can be approximated by a differential Riccati equation and a series of state-dependent Lyapunov equations, which can be solved recursively and lead to the closed-form state feedback control law . This control law can achieve semi-global asymptomatic stability of the closed-loop system. Numerical simulations verify that the proposed technique is compared favourably with the finite-time state-dependent Riccati equation (SDRE) technique in terms of computation efficiency and control effort, and also shows faster recovery of the quadrotor after the Cobra maneuver compared with conventional dynamics without the derivative of the rotation matrix.

Spatial Bee Optimized Network Theory (Sbont) Based Algorithm to Design of Dance Activity Venues in Elevating Tourism

Cultural tourism is a dynamic and enriching experience that involves exploring the unique heritage, traditions, arts, and customs of different societies. It goes beyond merely visiting landmarks; it delves into the heart and soul of a place, fostering understanding and appreciation for its cultural identity. Cultural tourism with a dance activity provides a vibrant and immersive way to experience the richness of a destination's cultural heritage. Participating in dance activities allows travelers to connect with local communities on a deeper level, learning about their traditions firsthand and forging meaningful connections. This paper proposes a novel approach for the spatial optimization of dance activity venues in cultural tourism destinations, integrating complex network theory with Spatial Bee Optimized Network Theory (SBONT). Cultural tourism thrives on offering diverse and immersive experiences, and dance activities play a pivotal role in enriching these experiences. However, optimizing the spatial layout of dance venues to enhance accessibility, cultural authenticity, and visitor satisfaction poses a complex problem. With integrating spatial considerations with network optimization principles, the proposed framework aims to maximize accessibility, connectivity, and cultural immersion for tourists while promoting sustainable development and resource utilization. Simulation results demonstrated that venues with a cultural significance score exceeding 0.8 experienced a 15% increase in visitor footfall compared to less significant counterparts. Moreover, the accessibility score for each venue improved by an average of 0.12, indicative of enhanced ease of access for tourists and local communities. These tangible outcomes underscore the algorithm’s effectiveness in orchestrating layouts that foster sustainable cultural tourism development while celebrating and preserving cultural heritage through dynamic dance activities.